United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
I nilid Stall-, Patent and Trademark Office 

Address: COMMISSIONER FOR PATENTS 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. CONFIRMATION NO. 



10/829.276 



04/22/2004 



65565 7590 02/01/201 

SUGHRUE-265550 

2100 PENNSYLVANIA AVE. NW 

WASHINGTON, DC 20037-3213 



Tomoki Ohkawa 



C ASCI II. RA, ANTONIO A 



PAPER NUMBER 



NOTIFICATION DATE | DELIVERY MODE 
02/01/2010 ELECTRONIC 



Please find below and/or attached an Office communication concerning this application or proceeding. 

The time period for reply, if any, is set in the attached communication. 

Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the 
following e-mail address(es): 

SUGHRUE265550@SUGHRUE.COM 
USPTO@SUGHRUE.COM 
PPROCESS INC, @ SUGHRl JE.COM 



PTOL-90A (Rev. 04/07) 



l/ffflrC? nVrliUli Otfff Iff ids y 


Application No. 

10/829,276 


Applicant(s) 

OHKAWA, TOMOKI 


Examiner 

Antonio A. Caschera 


Art Unit 

2628 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address — 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )KI Responsive to communication(s) filed on 19 January 2010 . 
2a )□ This action is FINAL. 2b)^ This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 
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DETAILED ACTION 

Continued Examination Under 37 CFR 1.114 

1 . Receipt is acknowledged of a request for continued examination under 37 CFR 1.114, 
including the fee set forth in 37 CFR 1.17(e) and a submission, filed on 01/19/10. 

Priority 

2. Acknowledgment is made of applicant's claim for foreign priority under 35 
U.S.C. 1 19(a)-(d). The certified copy has been filed in the pending application. 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness 
rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

3. Claims 1, 4-7 and 10 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Prandoni et al. (U.S. Patent 7,042,493) in view of Meier et al. (U.S. Patent 7,075,556). 

In reference to claim 1, Prandoni et al. discloses a video processing apparatus (see 
column 1, lines 44-58 and Figure 1) of a telecommunications system comprising: 

a video input section to which a video data comprising a plurality of frame images is 
input (see column 2, lines 9-14 and #101 and "video" of Figure #1 wherein Prandoni et al. 
discloses the apparatus to comprise of a background-foreground extraction module which 
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receives a video sequence. The Examiner interprets the background-foreground extraction 
module equivalent to the video input section since data as described in the claims is received by 
this module of Prandoni et al.) 

a drawing input section to which a drawn input image is input; 

an image control section that periodically cuts out a selected image in real time as static 
image information from the frame images at intervals of predetermined time (see column 2, lines 
9-14, 47-62, columns 2-3, lines 63-14 and column 5, lines 8-15 wherein Prandoni et al. discloses 
the apparatus to comprise of a background-foreground extraction module which receives a video 
sequence, say from a standard camera, and extracts one or several foreground features to appear 
in stroboscoping sequence. Prandoni et al. further discloses such extraction to occur based upon a 
number of "triggered" events one being frame-based triggering another being time-based 
triggering where foreground features are extracted at fixed time intervals, say every t seconds or 
as per user-defined triggering events. Lastly, Prandoni explicitly discloses the invention 
extended to on-demand video delivery systems and also sporting events both of which the 
Examiner interprets as providing the "real-time" environment for which the invention of 
Prandoni is disclosed as extended thereto.) and periodically extracts the drawn input image as 
input drawing static information at intervals of the predetermined time, wherein sound 
information from said frame images is output; 

an image information storage section that stores the static image information (see column 
5, lines 20-23 wherein Prandoni et al. claims a computer performing the methods of the 
invention. Note, the Examiner interprets the computer of Prandoni et al. to inherently comprise 
of some sort of memory for storing, at least temporarily, the foreground/background extracted 
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images/objects since Prandoni et al. discloses explicit manipulation of such information which 
would require at least temporary storage of the data.) and the input drawing static information; 

an image combining section that combines the static image information and the input 
drawing static information to create combined image information (see column 3, lines 15-41 and 
Figure 1 wherein Prandoni et al. discloses a synthesis module that combines selected foreground 
features and background visual information into a composite image); and 

an image drawing section that outputs the combined image information (see columns 3-4, 
lines 42-14 and Figure 1 wherein Prandoni et al. discloses a rendering module that in some 
instances, produces/outputs a video sequence of composite images) and; 

a display section that display the combined image information (see column 2, lines 38-42 
and #130 of Figure 1 wherein Prandoni et al. discloses a rendering module transforming data 
suitable for display on a video device), 

wherein the predetermined time is set to be greater than or equal to a processing time of 
the drawing processing apparatus, said processing time being a duration starting from a time 
when the selected image is cut out to a time when the combined image information is displayed 
on the display section (see column 3, lines 3-4, 11-13 and 62-67 wherein Prandoni et al. 
discloses extraction to occur based upon a number of "triggered" events one being frame-based 
triggering another being time-based triggering and another being user-defined where foreground 
features are extracted at fixed time intervals, say every t seconds and as per a user's involvement 
clicking on image frames. Prandoni et al. further discloses the video sequence being rendered 
when the triggering events are reached. Therefore, the Examiner interprets the above mentioned 
per frame triggered extraction techniques of Prandoni et al. at least equal to the time from when 
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an image is selected via the frame-defined trigger event to the time that a composite image is 
rendered/displayed. Further, the Examiner states that such a timing as recited by Applicant's 
claim is inherent to the timing of Prandoni et al. since it is inherent that video sequence data is 
displayed at x frames per second and therefore, the time between when extraction is performed 
for a frame and when combined information is displayed is predetermined.) and 

wherein said sound information and the combined image information are output. 

Prandoni et al. does not explicitly disclose extracting input drawing static information 
from a drawn input image every predetermined time. Meier et al. discloses a system for 
annotating video or still images wherein a user utilizing a telestrator to draw upon video images 
and to combine the drawn and video images for broadcasted display (see column 1, lines 23-24, 
column 3, lines 60-63, columns 4-5, lines 51-4). Meier et al. explicitly discloses a telestration 
processor receiving an input from a touch screen or pen/stylus for a drawing upon the screen (see 
column 4, lines 5 1-60 and column 12, lines 41-44). Meier et al. discloses the telestration 
processor outputting such pixel data along with time code data associating such pixel data with 
specific frames/fields to a rendering processor which processes such data on a frame by frame 
basis (see columns 15-16, lines 58-2 and Figure 1 1). Note, the Examiner interprets such reading 
of data from the touch screen/pen/stylus input via the telestration processor, marking such data 
with specific time code data, more specifically the time code data being in units of frames/fields, 
equivalent to Applicant's periodically extracting the drawing input at intervals of a 
predetermined time (as indicated by the associated time code in Meier et al). Meier et al. 
discloses capturing video image using cameras and sending the additional data in the camera's 
audio channel (see column 5, lines 19-32). Meier et al. also discloses the camera to comprise of 
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a microphone (see column 6, line 55-60). The Examiner interprets that since Meier et al. 
explicitly discloses the camera to comprise of a microphone, that an audio channel is utilized in 
sending data and further that Meier et al.'s entire invention deals with telestration display 
processing, the audio channel or "sound information" is at least inherently output as per the 
framed drawings of the combined video and drawing data. It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to implement the telestration system 
including touch screen/pen/stylus and video based image processing techniques of Meier et al. 
with the foreground/background image data processing techniques of Prandoni et al. in order to 
allow the viewing of real-time user interaction data along with video captured data on a display 
simultaneously allowing for multiple viewers/users of the system to better graphically articulate 
information to one another that is explicit to the captured video (see column 1 , lines 40-54 and 
column 2, lines 30-32 and 56-63 of Meier et al.). (see Response to Arguments below) 
In reference to claim 4, Prandoni et al. and Meier et al. disclose all of the claim 
limitations as applied to claim 1 above. Prandoni et al. discloses the apparatus to comprise of a 
background-foreground extraction module which receives a video sequence, say from a standard 
camera, and extracts one or several foreground features to appear in stroboscoping sequence (see 
column 2, lines 9-14, 47-62 and columns 2-3, lines 63-14). Prandoni et al. discloses a rendering 
module that in some instances, produces/outputs a video sequence of composited images (see 
columns 3-4, lines 42-14 and Figure 1), the rendering module seen as providing equivalent 
functionality as Applicant's moving image reproducing section. Prandoni et al. claims a 
computer performing the methods of the invention (see column 5, lines 20-23). Note, the 
Examiner interprets the computer of Prandoni et al. to inherently comprise of some sort of 
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memory or moving image storage section for storing, at least temporarily, the rendered/output 
video stroboscoped sequences. 

In reference to claim 5, Prandoni et al. and Meier et al. disclose all of the claim 
limitations as applied to claim 1 above. Prandoni et al. discloses the apparatus to comprise of a 
background-foreground extraction module which receives a video sequence, say from a standard 
camera, and extracts one or several foreground features to appear in stroboscoping sequence (see 
column 2, lines 9-14, 47-62 and columns 2-3, lines 63-14). Prandoni et al. discloses a rendering 
module that in some instances, produces/outputs a video sequence of composited images (see 
columns 3-4, lines 42-14 and Figure 1). Meier et al. discloses allowing the user to create a 
drawing defined by polygons, alpha values, color values, texture values and fill characteristics 
(see columns 15-16, liens 58-1) and in one particular embodiment, details the user drawing a 
arrow/line which the Examiner interprets as inherently equivalent to vector data (see Figures 1-4 
and column 12, lines 41-43). Meier et al. also discloses combining such drawing input with 
video frame data capture from cameras (see columns 15-16, lines 58-8 and Figures 11-12). It 
would have been obvious to one of ordinary skill in the art at the time the invention was made to 
implement the telestration system including touch screen/pen/stylus and video based image 
processing techniques of Meier et al. with the foreground/background image data processing 
techniques of Prandoni et al. in order to allow the viewing of real-time user interaction data 
along with video captured data on a display simultaneously allowing for multiple viewers/users 
of the system to better graphically articulate information to one another that is explicit to the 
captured video (see column 1, lines 40-54 and column 2, lines 30-32 and 56-63 of Meier et al). 
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In reference to claim 6, Prandoni et al. and Meier et al. disclose all of the claim 
limitations as applied to claim 5 above. Meier et al. discloses the drawing data to include pixel 
coordinates for the drawn polygons, color data, tessellations per quadrilateral (number of 
subdivides), texture data and fill characteristics for the polygons (see columns 15-16, lines 65-2) 
which the Examiner interprets equivalent to Applicant's drawing data. 

In reference to claim 7, claim 7 is equivalent in scope to claim 1 and is therefore rejected 
under similar rationale. Further, it is noted that claim 7 recites a method type claim, the steps of 
the method as claimed are disclosed by the above recited teachings of the combination of 
Prandoni et al. and Meier et al. 

In reference to claim 10, claim 10 is equivalent in scope to claim 1 and is therefore 
rejected under similar rationale. Further, it is noted that claim 10 recites a computer-readable 
medium/program type claim, the steps of the program executed by the computer as claimed are 
disclosed by the above recited teachings of the combination of Prandoni et al. and Meier et al.. 
Prandoni et al. claims a computer performing the methods of the invention (see column 5, lines 
20-23). Note, the Examiner interprets the computer of Prandoni et al. to inherently comprise of 
some type of memory or medium for storing instructions/programs to execute or perform tasks. 
4. Claims 3 and 9 are rejected under 35 U.S.C. 103(a) as being unpatentable over Prandoni 
et al. (U.S. Patent 7,042,493), Meier et al. (U.S. Patent 7,075,556) and further in view of 
Seedholm, Peter. "Print Screen Button Tutorial." 
(http://www.ibiblio.org/virtualcell/Tutorl/TandR/prtscr.html). 

In reference to claims 3 and 9, Prandoni et al. and Meier et al. disclose all of the claim 
limitations as applied to claims 1 and 7 respectively above. Prandoni et al. claims a computer 
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performing the methods of the invention (see column 5, lines 20-23) which the Examiner 
interprets as inherently comprising a keyboard providing some sort of input to the invention. 
Neither Prandoni et al. nor Meier et al. explicitly disclose the image drawing section having a 
function of capturing the combined image using a screen capture signal by an image capture 
operation of a user. Seedholm discloses a tutorial for capturing and pasting a displayed screen of 
data using the print screen button on a keyboard (see steps 1-6 of Seedholm). It would have been 
obvious to one of ordinary skill in the art at the time the invention was made to implement the 
print screen display screen capturing techniques of Seedholm with the telestration system of 
Meier et al. and foreground/background image data processing techniques of Prandoni et al. in 
order to capture the current displayed image on a display monitor and further process the image 
by i.e. pasting it into another program/application, transmitting it over a network medium, saving 
it to a computer medium etc. 

5. Claim 1 1 is rejected under 35 U.S.C. 103(a) as being unpatentable over Prandoni et al. 
(U.S. Patent 7,042,493), Meier et al. (U.S. Patent 7,075,556) and further in view of Dempski et 
al. (U.S. Pub 20040155902). 

In reference to claim 11, Prandoni et al. discloses a video processing apparatus (see 
column 1, lines 44-58 and Figure 1) teleconference system in which a plurality of participant 
terminals which participate in a conference are connected through a communication line, 
wherein a drawing processing apparatus comprises: 

a video input section to which a video data comprising a plurality of frame images is 
input (see column 2, lines 9-14 and #101 and "video" of Figure #1 wherein Prandoni et al. 
discloses the apparatus to comprise of a background-foreground extraction module which 
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receives a video sequence. The Examiner interprets the background-foreground extraction 
module equivalent to the video input section since data as described in the claims is received by 
this module of Prandoni et al.) 

a drawing input section to which a drawn input image is input; 

an image control section that periodically cuts out a selected image in real time as static 
image information from the frame images at intervals of predetermined time (see column 2, lines 
9-14, 47-62, columns 2-3, lines 63-14 and column 5, lines 8-15 wherein Prandoni et al. discloses 
the apparatus to comprise of a background-foreground extraction module which receives a video 
sequence, say from a standard camera, and extracts one or several foreground features to appear 
in stroboscoping sequence. Prandoni et al. further discloses such extraction to occur based upon a 
number of "triggered" events one being frame-based triggering another being time-based 
triggering where foreground features are extracted at fixed time intervals, say every t seconds or 
as per user-defined triggering events. Lastly, Prandoni explicitly discloses the invention 
extended to on-demand video delivery systems and also sporting events both of which the 
Examiner interprets as providing the "real-time" environment for which the invention of 
Prandoni is disclosed as extended thereto.) and periodically extracts the drawn input image as 
input drawing static information at intervals of the predetermined time, wherein sound 
information from said frame images is output; 

an image information storage section that stores the static image information (see column 
5, lines 20-23 wherein Prandoni et al. claims a computer performing the methods of the 
invention. Note, the Examiner interprets the computer of Prandoni et al. to inherently comprise 
of some sort of memory for storing, at least temporarily, the foreground/background extracted 
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images/objects since Prandoni et al. discloses explicit manipulation of such information which 
would require at least temporary storage of the data.) and the input drawing static information; 

an image combining section that combines the static image information and the input 
drawing static information to create combined image information (see column 3, lines 15-41 and 
Figure 1 wherein Prandoni et al. discloses a synthesis module that combines selected foreground 
features and background visual information into a composite image); and 

an image drawing section that outputs the combined image information (see columns 3-4, 
lines 42-14 and Figure 1 wherein Prandoni et al. discloses a rendering module that in some 
instances, produces/outputs a video sequence of composite images) and; 

a display section that display the combined image information (see column 2, lines 38-42 
and #130 of Figure 1 wherein Prandoni et al. discloses a rendering module transforming data 
suitable for display on a video device), 

wherein the predetermined time is set to be greater than or equal to a processing time of 
the drawing processing apparatus, said processing time being a duration starting from a time 
when the selected image is cut out to a time when the combined image information is displayed 
on the display section (see column 3, lines 3-4, 11-13 and 62-67 wherein Prandoni et al. 
discloses extraction to occur based upon a number of "triggered" events one being frame-based 
triggering another being, another being time-based triggering and another being user-defined 
where foreground features are extracted at fixed time intervals, say every t seconds and as per a 
user's involvement clicking on image frames. Prandoni et al. further discloses the video 
sequence being rendered when the triggering events are reached. Prandoni et al. further discloses 
the video sequence being rendered when the triggering events are reached. Therefore, the 
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Examiner interprets the above mentioned per frame triggered extraction techniques of Prandoni 
et al. at least equal to the time from when an image is selected via the frame-defined trigger 
event to the time that a composite image is rendered/displayed. Further, the Examiner states that 
such a timing as recited by Applicant's claim is inherent to the timing of Prandoni et al. since it 
is inherent that video sequence data is displayed at x frames per second and therefore, the time 
between when extraction is performed for a frame and when combined information is displayed 
is predetermined.), and 

wherein said sound information and the combined image information are output. 

Prandoni et al. does not explicitly disclose extracting input drawing static information 
from a drawn input image every predetermined time. Meier et al. discloses a system for 
annotating video or still images wherein a user utilizing a telestrator to draw upon video images 
and to combine the drawn and video images for broadcasted display (see column 1, lines 23-24, 
column 3, lines 60-63, columns 4-5, lines 51-4). Meier et al. explicitly discloses a telestration 
processor receiving an input from a touch screen or pen/stylus for a drawing upon the screen (see 
column 4, lines 51-60 and column 12, lines 41-44). Meier et al. discloses the telestration 
processor outputting such pixel data along with time code data associating such pixel data with 
specific frames/fields to a rendering processor which processes such data on a frame by frame 
basis (see columns 15-16, lines 58-2 and Figure 1 1). Note, the Examiner interprets such reading 
of data from the touch screen/pen/stylus input via the telestration processor, marking such data 
with specific time code data, more specifically the time code data being in units of frames/fields, 
equivalent to Applicant's periodically extracting the drawing input at intervals of a 
predetermined time (as indicated by the associated time code in Meier et al). Meier et al. 
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discloses capturing video image using cameras and sending the additional data in the camera's 
audio channel (see column 5, lines 19-32). Meier et al. also discloses the camera to comprise of 
a microphone (see column 6, line 55-60). The Examiner interprets that since Meier et al. 
explicitly discloses the camera to comprise of a microphone, that an audio channel is utilized in 
sending data and further that Meier et al.'s entire invention deals with telestration display 
processing, the audio channel or "sound information" is at least inherently output as per the 
framed drawings of the combined video and drawing data. It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to implement the telestration system 
including touch screen/pen/stylus and video based image processing techniques of Meier et al. 
with the foreground/background image data processing techniques of Prandoni et al. in order to 
allow the viewing of real-time user interaction data along with video captured data on a display 
simultaneously allowing for multiple viewers/users of the system to better graphically articulate 
information to one another that is explicit to the captured video (see column 1, lines 40-54 and 
column 2, lines 30-32 and 56-63 of Meier et al.). Although Meier et al. does disclose such 
telestration techniques used in a telecommunications environment, neither Prandoni et al. nor 
Meier et al. explicitly disclose their image data processing techniques within a specific 
teleconferencing environment however Dempski et al. does. Dempski et al. discloses a computer 
system in a teleconferencing environment for superimposing a computer-generated image onto a 
video image or vice versa (see paragraphs 5 and 1 1 of Dempski et al.). It would have been 
obvious to one of ordinary skill in the art at the time the invention was made to implement the 
teleconference superimposing techniques of Dempski et al. with the telestration system including 
touch screen/pen/stylus and video based image processing techniques of Meier et al. and 
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foreground/background image data processing techniques of Prandoni et al. in order to allow 
multiple users to view, manipulate, and share real-time data in a teleconferencing environment 
making it easier to communicate to users in remote locations (see paragraphs 2-3 of Dempski et 
al). 

Response to Arguments 

6. Applicant's arguments filed 01/19/10 have been fully considered but they are not 
persuasive. 

In reference to the claims, Applicant argues that none of the applied prior art teaches the 
newly amended limitation of the claims of, "sound information from said frame images is 
output" and "wherein said sound information and the combined image information are output" 
(see page 7 of Applicant's Remarks). In response, the Examiner disagrees. Meier et al. discloses 
capturing video image using cameras and sending the additional data in the camera's audio 
channel (see column 5, lines 19-32). Meier et al. also discloses the camera to comprise of a 
microphone (see column 6, line 55-60). The Examiner interprets that since Meier et al. explicitly 
discloses the camera to comprise of a microphone, that an audio channel is utilized in sending 
data and further that Meier et al.'s entire invention deals with telestration display processing, the 
audio channel or "sound information" is at least inherently output as per the framed drawings of 
the combined video and drawing data. The Examiner believes at least Meier et al. of the cited 
prior art to teach such an amended limitation and therefore maintains the rejection based upon 
Prandoni et al. and Meier et al. 
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Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Antonio Caschera whose telephone number is (571) 272-7781. 
The examiner can normally be reached Monday, Tuesday, Thursday and Friday between 7:00 
AM and 5:30 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kee Tung, can be reached at (571) 272-7794. 

Any response to this action should be mailed to: 

Commissioner of Patents and Trademarks 
Washington, D.C. 20231 

or faxed to: 

571-273-8300 (Central Fax) 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the Technology Center 2600 Customer Service Office whose telephone 
number is (571)272-2600. 

/Antonio A Caschera/ 

Primary Examiner, Art Unit 2628 

1/28/10 



